Decoding High-Dimensional Data: Linear Dimensionality Reduction Techniques Revisited

Abstract

In our rapidly digitizing world, data is being collected at a never-before-seen pace from dynamic global sectors like healthcare, manufacturing, sales, IoT devices, the web, smart gadgets, social media, and organizations on a regular basis. The properties of this type of data are high dimensionality, large volume, redundant features, and noise. As such, representing and processing such a large amount of complex, heterogeneous data becomes much more challenging. In many sectors where there is a lot of data with lots of columns or classes, dimensionality reduction techniques are crucial. One essential method for evaluating and understanding high-dimensional data is the application of dimensionality reduction techniques.These methods collect a wealth of interesting data properties, including dynamical structure, correlation between data sets, covariance, and input-output interactions. This paper reviews several types of algorithms used in dimension reduction in order to provide readers with a thorough and lucid summary of the field and to give them a sense of how to assess its increasing importance over the past few years. Since linear dimensionality reduction techniques have straightforward geometric interpretations and generally appealing computing features, they are fundamental to the analysis of high dimensional data. Principal Component Analysis (PCA), Singular Value Decomposition (SVD), linear discriminant analysis(LDA), and Independent Component Analysis (ICA) are the four linear dimensionality reduction approaches that will be examined and compared in this study. The purpose of this study is to examine and contrast the benefits and drawbacks of dimensionality reduction in several widely applied mathematical concepts and techniques.

Introduction

With the rapid growth of digital data from social media, healthcare, finance, and other fields, datasets have become extremely large, complex, and high-dimensional. This creates the “curse of dimensionality,” where machine learning models become slower, less accurate, and harder to train due to redundant, noisy, and irrelevant features. High-dimensional data also increases storage requirements and reduces visualization and interpretability.

To solve this, the text emphasizes dimensionality reduction (DR), which reduces the number of features while preserving important information. DR improves computational efficiency, reduces noise, prevents overfitting, enhances accuracy, and enables easier visualization.

The paper focuses on linear dimensionality reduction techniques, including:

• PCA (Principal Component Analysis)
• SVD (Singular Value Decomposition)
• LDA (Linear Discriminant Analysis)
• ICA (Independent Component Analysis)

Among these, PCA is discussed in detail. PCA transforms correlated variables into a smaller set of uncorrelated components called principal components, capturing maximum variance in the data. It is widely used due to its simplicity, efficiency, and effectiveness in reducing dimensionality while preserving key patterns. The literature shows PCA has applications in image analysis, signal processing, time-series data, and word embeddings.

The text also explains SVD, a matrix factorization technique closely related to PCA. SVD decomposes data into simpler matrices to extract meaningful structure, reduce noise, and support tasks like compression, classification, clustering, and feature extraction.

Conclusion

The increasing amount of high-dimensional data makes it necessary to employ dimensionality reduction techniques to make the data easier to handle. This paper provides a detailed review of the different linear dimensionality reduction techniques that are used, identifies the various approaches that are used to lower the dimensions in order to increase the accuracy of algorithms, and provides guidance on how to evaluate the field\'s recent surge in popularity. It is crucial to display the high dimensions of data in lesser dimensions without sacrificing the data\'s uniqueness. But regardless of the type of data we are dealing with, choosing the appropriate strategy for a particular set of data is essential and greatly influenced by the kind of data we are dealing with. The benefits and drawbacks of each technique are also explained in this study. Many methodologies such as Principal Component Analysis (PCA), Singular Value Decomposition (SVD), Linear Discriminant Analysis (LDA), Independent Component Analysis (ICA) are analysed and compared based on various attributes. Conclusively, all these methods seek to provide the appropriate information while minimizing complexity. Additionally, this work provides a reasonable starting point for comparing various dimensional approaches.

References

[1] Tang, W., & Zhong, S. (2007). Pairwise constraints-guided dimensionality reduction. Computational Methods of Feature Selection, 295-312 [2] Donoho, D.L. (2000) High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality. Lecture Delivered at the “Mathematical Challenges of the 21st Century” Conference of the American Math. Society, Los Angeles. http://www-stat.stanford.edu/donoho/Lectures/AMS2000/AMS2000.html . [3] F. Anowar and S. Sadaoui, “Incremental Neural-Network Learning for Big Fraud Data,” in 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2020, pp. 3551–3557, doi: 10.1109/SMC42975.2020.9283136. [4] Jain, R., & Xu, W. (2021). RHDSI: a novel dimensionality reduction based algorithm on high dimensional feature selection with interactions. Information Sciences, 574, 590-605. [5] L. J. P. Van Der Maaten, E. O. Postma, and H. J. Van Den Herik, “Dimensionality Reduction: A Comparative Review,” J. Mach. Learn. Res., vol. 10, pp. 1–41, 2009, doi: 10.1080/13506280444000102. [6] Wei Wei, Member, IEEE, Qin Yue, Kai Feng, Junbiao Cui, and Jiye Liang, Senior Member, IEEE, “Unsupervised Dimensionality Reduction based on fusing multiple clustering results,” IEEE Transactions on Knowledge and Data Engineering, 21 September 2021. [7] K. Fukunaga, Introduction to Statistical Pattern Recognition. San Diego, CA, USA, 1990. [8] Laurens van der Maaten “An Introduction to Dimensionality Reduction Using Matlab”, MICC, Maastricht University. [9] Verleysen M., François D. (2005) The Curse of Dimensionality in Data Mining and Time Series Prediction. In: Cabestany J., Prieto A., Sandoval F. (eds) Computational Intelligence and Bioinspired Systems. IWANN 2005. Lecture Notes in Computer Science, vol 3512. Springer, Berlin, Heidelberg. [10] Thippa Reddy Gadekallu, Praveen Kumar Reddy, Kuruva Lakshman, Rajesh Kaluri, “Analysis of Dimensionality Reduction Techniques on Big Data,” IEEE Access (Vol: 8) 16 March 2020. [11] Zebari , R., Abdulazeez, A., Zeebaree, D., Zebari, D., & Saeed, J. (2020). A Comprehensive Review of Dimensionality Reduction Techniques for Feature Selection and Feature Extraction. Journal of Applied Science and Technology Trends, 1(2), 56 – 70 [12] G.N.Ramadevi and K.Usharani- “Study On Dimensionality Reduction Techniques And Applications” - 2- Vol 04, Special Issue01; 2013 -Publications Of Problems & Application In Engineering Research – Paper [13] Juvonen, A., Sipola, T., Hämäläinen, T. (2015). Online anomaly detection using dimensionality reduction techniques for HTTP log analysis, Comput. Networks, vol. 91, pp. 46–56, doi: 10.1016/j.comnet.2015.07.019. [14] Liu, L., Zsu, M. T. (2009). Encyclopedia of Database Systems, Springer Publishing Company [15] Christopher J. C. Burges “Dimension Reduction: A Guided Tour-Foundations and TrendsR_ in Machine Learning” Vol. 2, No. 4 (2009) 275–365 _c 2010 DOI: 10.1561/2200000002. [16] L. O. Jimenez and D. A. Landgrebe, “Supervised classification in high-dimensional space: geometrical, statistical, and asymptotical properties of multivariate data,” IEEE Trans. Syst. Man, Cybern. Part C (Applications Rev., vol. 28, no. 1, pp. 39–54, 1998, doi: 10.1109/5326.661089. [17] F. Salo, A. B. Nassif, and A. Essex, “Dimensionality reduction with IG-PCA and ensemble classifier for network intrusion detection,” Comput. Networks, vol. 148, pp. 164–175, Jan. 2019, Accessed: Sep. 03, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pi i/S1389128618303037. [18] M. Partridge and C. Rafael, “Fast dimensionality reduction and simple PCA,” Intell. Data Anal., vol. 2, no. 3, pp. 292–298, 1998, doi: 10.3233/IDA-1998-2304. [19] K. V Ravi Kanth, D. Agrawal, A. El Abbadi, and A. Singh, “Dimensionality Reduction for Similarity Searching in Dynamic Databases,” Comput. Vis. Image Underst., vol. 75, no. 1, pp. 59–72, 1999, doi: https://doi.org/10.1006/cviu.1999.0762. [20] P. Switzer, “Extensions of linear discriminant analysis for statistical classification of remotely sensed satellite imagery,” J. Int. Assoc. Math. Geol., vol. 12, no. 4, pp. 367–376, 1980, doi: 10.1007/BF01029421. [21] Hyvärinen A, Oja E (2000) Independent component analysis: algorithms and applications. Neural Netw 13(4–5):411–430 [22] S. Khalid, T. Khalil, and S. Nasreen, “A survey of feature selection and feature extraction techniques in machine learning,” in 2014 Science and Information Conference, 2014, pp. 372–378, doi: 10.1109/SAI.2014.6918213. [23] A. B. Nassif, M. Azzeh, L. F. Capretz, and D. Ho, “A comparison between decision trees and decision tree forest models for software development effort estimation,” in 2013 3rd International Conference on Communications and Information Technology, ICCIT 2013, 2013, pp. 220–224, doi: 10.1109/ICCITechnology.2013.6579553. [24] M. Azzeh “Analogy-based effort estimation: a new method to discover set of analogies from dataset characteristics,” IET Softw., vol. 9, no. 2, pp. 39–50, 2015, doi: 10.1049/ietsen.2013.0165. [25] M. Azzeh, S. Banitaan, and F. Almasalha, “Pareto efficient multi-objective optimization for local tuning of analogy-based estimation,” Neural Comput. Appl., vol. 27, no. 8, pp. 2241– 2265, 2016, doi: 10.1007/s00521-015-2004-y. [26] M. Ramaswami and R. Bhaskaran, “A Study on Feature Selection Techniques in Educational Data Mining”, Journal of Computing, Vol. 1, No. 1, pp. 7-11, 2009. [27] J.E. Jackson. A User’s Guide to Principal Components. New York: John Wiley and Sons, 1991. [28] I. T. Jolliffe. Principal Component Analysis. Sprin, second edition, 2002. [29] S. Wold. Principal Component Analysis. Chemometrics and Intelligent Laboratory Systems, 2:37–52, 1987. [30] R. Vidal, Y. Ma, and S. S. Sastry. Generalized Principal Component Analysis. Springer, 2016 [31] I. T. Jolliffe and J. Cadima. Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A, page 20150202, 2016. [32] I.T. Jolliffe. Principal Component Analysis. Springer-Verlag, 1986. [33] K.V. Mardia, J.T. Kent, and J.M. Bibby. Multivariate Analysis. Probability and Mathematical Statistics. Academic Press, 1995. [34] Person, K. (1901) On Lines and Planes of Closest Fit to System of Points in Space. Philiosophical Magazine, 2, 559-572. http://dx.doi.org/10.1080/14786440109462720 [35] Jenkins, O.C. and Mataric, M.J. (2002) Deriving Acion and Behavior Primitives from Human Motion Data. International Conference n Robots and Systems, 3, 2551-2556. [5] Jain, A.K. and Dubes, R.C. (1962) Algorithms for Clastering Data. Prentice Hall, Upper Saddle River. [36] Mardia, K.V., Kent, J.T. and Bibby, J.M. (1995) Multivariate Analysis Probability and Mathematical Statistics. Academic Press, Waltham. [37] (2002) Francesco Camastra Data Dimensionality Estimation Methods, a Survey INFM-DISI, University of Genova, Genova. [38] Fukunaga, K. (1982) Intrinsic Dimensionality Extraction, in Classification, Pattern Recognition and Reduction of Dimensionality, Vol. 2 of Handbook of Statistics, North Holland, 347-362. [39] Roweis, S.T. (1997) EM Algorithms for PCA and SPCA. Advances in Neural Information Processing Systems, 10, 626-632. [40] Lawrence, N.D. (2005) Probabilistic Non-Linear Proncipal Component Analysis with Gaussian Process Latent Variable Models. Journal of Machine Learning Research, 6, 1783-1816. [41] Welling, M., Rosen-Zvi, M. and Hinton, G. (2004) Exponential Family Harmoniums with an Application to Information Retrieval. Advances in Neural Information Processing Systems, 17, 1481-1488. [42] Turk, M.A. and Pentland, A.P. (1991) Face Recognition Using Eigenfaces. Proceedings of the Computer Vision and Pattern Recognition 1991, Maui, 586-591. http://dx.doi.org/10.1109/CVPR.1991.139758 [43] Huber, R., Ramoser, H., Mayer, K., Penz, H. and Rubik, M. (2005) Classification of Coins Using an Eigenspace Approach. Pattern Recognition Letters, 26, 61-75. http://dx.doi.org/10.1016/j.patrec.2004.09.006 [44] Posadas, A.M., Vidal, F., de Miguel, F., Alguacil, G., Pena, J., Ibanez, J.M. and Morales, J. (1993) Spatialtemporal Analysis of a Seismic Series Using the Principal Components Method. Journal of Geophysical Research, 98, 1923-1932. http://dx.doi.org/10.1029/92JB02297 [45] H. Abdi and L. J. Williams, “Principal Component Analysis,” Wiley Interdiscip. Rev. Comput. Stat., vol. 4, no. 2, pp. 433–459, 2010 [46] Arunasakthi, K., &KamatchiPriya, L. (2014). A review on linear and non-linear dimensionality reduction techniques. Machine Learning and Applications: An International Journal (MLAIJ), 1(1), 65-76. [47] Yan, S., Xu, D., Zhang, B., Zhang, H. J., Yang, Q., & Lin, S. (2006). Graph embedding and extensions: A general framework for dimensionality reduction. IEEE transactions on pattern analysis and machine intelligence, 29(1), 40-51. [48] Raunak, V. (2017). Simple and Effective Dimensionality Reduction for Word Embeddings, arXiv, no. Nips, pp. 1–6. [49] Jiang, J., Ma, J., Chen, C., Wang, Z., Cai, Z. and Wang, L. (2018). SuperPCA: A Superpixelwise PCA Approach for Unsupervised Feature Extraction of Hyperspectral Imagery, IEEE Trans. Geosci. Remote Sens., vol. 56, no. 8, pp. 4581–4593, doi: 10.1109/TGRS.2018.2828029 [50] N. Verbeeck, R. M. Caprioli, and R. Van de Plas, “Unsupervised machine learning for exploratory data analysis in imaging mass spectrometry,” Mass Spectrometry Reviews, vol. 39, no. 3, pp. 245–291, 2020. [51] L. J. Al-Omairi, J. Abawajy, M. U. Chowdhury, and T. AlQuraishi, “An empirical analysis of graph-based linear dimensionality reduction techniques,” Concurrency and Computation: Practice and Experience, vol. 33, no. 5, pp. 1–13, 2021. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.5990 [52] T. Howley, M. G. Madden, M.-L. O’Connell, and A. G. Ryder, “The effect of principal component analysis on machine learning accuracy with high dimensional spectral data,” in Applications and Innovations in Intelligent Systems XIII. Springer London, pp. 209–222. [Online]. Available: https://doi.org/10.1007/1-84628-224-116 [53] D. Cao, Y. Tian, and D. Bai, “Time series clustering method based on principal component analysis,” in Proceedings 5th International Conference on Information Engineering for Mechanics and Materials (ICIMM 2015). Citeseer, 2015, pp. 888–895 [54] F. Alshammri and J. Pan, “Moving dynamic principal component analysis for non-stationary multivariate time series,” Computational Statistics, vol. 36, no. 3, pp. 1–41, Mar. 2021. [Online]. Available: https://doi.org/10.1007/s00180-021- 01081-8 [55] P. Tanisaro and G. Heidemann, “Dimensionality reduction for visualization of time series and trajectories,” in Image Analysis. Springer International Publishing, 2019, pp. 246–257. [Online]. Available: https://doi.org/10.1007/978- 3-030-20205-721 [56] [55] A. Widodo and I. Budi, “Model selection using dimensionality reduction of time series characteristics,” in International Symposium on Forecasting, Seoul, South Korea, 2013, pp. 57–118. [57] [56] G. Gawde and J. Pawar, “Shape based time series reduction using pca,” in 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), 2017, pp. 1–4. [58] F. Anowar, S. Sadaoui, and B. Selim, “Conceptual and empirical comparison of dimensionality reduction algorithms (pca, kpca, lda, mds, svd, lle, isomap, le, ica, t-sne),” Computer Science Review, vol. 40, pp. 1–13, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1574013721000186. [59] N. Sharma, “Principal component analysis,” https://heartbeat.fritz.ai/understanding-the-mathematics-behind-principalcomponent-analysis-efd7c9ff0bb3, 2021, last accessed 21 December 2021. [60] J. Shlens, “A tutorial on principal component analysis,” arXiv preprint arXiv:1404.1100, pp. 1–12, 2014 [61] Y. Jaradat, M. Masoud, I. Jannoud, A. Manasrah, and M. Alia, “A Tutorial on Singular Value Decomposition with Applications on Image Compression and Dimensionality Reduction,” 2021 International Conference on Information Technology (ICIT), 2021, pp. 769-772, doi: 10.1109/ICIT52682.2021.9491732. [62] P. David. (2015) \"Linear Algebra: A Modern Introduction\", 4th ed. Cengage Learning [63] L. J. Sciacca and R. J. Evans, “Multidimensional inverse problems in ultrasonic imaging,” in Multidimensional Systems: Signal Processing and Modeling Techniques, ser. Control and Dynamic Systems, C. Leondes, Ed. Academic Press, 1995, vol. 69, pp. 1–48. [Online]. Available:https://www.sciencedirect.com/science/article/pii/S0090526705800374 [64] S. Brunton and J. Kutz, Singular Value Decomposition (SVD). Cambridge University Press, 02 2019, pp. 3–46. [65] M. Frank and J. M. Buhmann, “Selecting the rank of truncated SVD by maximum approximation capacity,” in 2011 IEEE International Symposium on Information Theory Proceedings. IEEE, Jul. 2011, pp. 1036–1040. [Online]. Available: https://doi.org/10.1109/isit.2011.6033687 [66] A. Dutta, J. Liang, and X. Li, “A fast and adaptive svd-free algorithm for general weighted low-rank recovery,” arXiv preprint arXiv:2101.00749, pp. 1–27, 2021 [67] F. Anowar, S. Sadaoui, and B. Selim, “Conceptual and empirical comparison of dimensionality reduction algorithms (pca, kpca, lda, mds, svd, lle, isomap, le, ica, t-sne),” Computer Science Review, vol. 40, pp. 1–13, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1574013721000186 [68] Qiao, Hanli. (2015) \"New SVD based initialization strategy for non-negative matrix factorization\" Pattern Recognition Letters 63: 71-77. [69] Taufik faudi, bustami yusuf, Munzir, “Singular value Decomposition for dimensionality reduction in unsupervised text learning problems”, ieee conference on Education tech and computer 29th july 2010. [70] Kumar, Manoj; and Ankita Vaish. (2017) \"An efficient encryption-then-compression technique for encrypted images using SVD.\" Digital Signal Processing 60: 81-89 [71] J. C. S. de Souza, T. M. L. Assis, and B. C. Pal, “Data compression in smart distribution systems via singular value decomposition,” IEEE Transactions on Smart Grid, vol. 8, no. 1, pp. 275–284, 2017. [ [72] D. Kaur, G. S. Aujla, N. Kumar, A. Y. Zomaya, C. Perera, and R. Ranjan, “Tensor-based big data management scheme for dimensionality reduction problem in smart grid systems: Sdn perspective,” IEEE Transactions on Knowledge and Data Engineering, vol. 30, no. 10, pp. 1985–1998, Oct 2018. [73] A. Zhang and D. Xia, “Tensor svd: Statistical and computational limits,” IEEE Transactions on Information Theory, 2018. [74] L. T. Yang, X. Wang, X. Chen, L. Wang, R. Ranjan, X. Chen, and M. J. Deen, “A multi-order distributed hosvd with its incremental computing for big services in cyber-physical-social systems,” IEEE Transactions on Big Data, 2018. [75] I. Kajo, N. Kamel, and Y. Ruichek, “Incremental tensor-based completion method for detection of stationary foreground objects,” IEEE Transactions on Circuits and Systems for Video Technology, 2018 [76] F. Fallucchi and F. M. Zanzotto, “Singular value decomposition for feature selection in taxonomy learning,” in Proceedings of the International Conference RANLP-2009. Borovets, Bulgaria: Association for Computational Linguistics, Sep. 2009, pp. 82–87. [Online]. Available: https://www.aclweb.org/anthology/R09-1016 [77] I. Bowala and M. Fernando, “A novel model for Time-Series Data Clustering Based on piecewise SVD and BIRCH for Stock Data Analysis on Hadoop Platform,” Advances in Science, Technology and Engineering Systems Journal, vol. 2, no. 3, pp. 855–864, 2017. [Online]. Available: http://astesj.com/v02/i03/p106/ [78] W. Lin, J. Z. Huang, and T. McElroy, “Time series seasonal adjustment using regularized singular value decomposition,” Journal of Business & Economic Statistics, vol. 38, no. 3, pp. 487–501, feb 2019. [Online]. Available: https://doi.org/10.1080/07350015.2018.1515081 [79] H. Li, C. Lin, X. Wan, and Z. xin Li, “Feature representation and similarity measure based on covariance sequence for multivariate time series,” IEEE Access, vol. 7, pp. 67 018–67 026, 2019. [80] V. Choqueuse, P. Granjon, A. Belouchrani, F. Auger, and M. Benbouzid, “Monitoring of three-phase signals based on singular-value decomposition,” IEEE Transactions on Smart Grid, vol. PP, pp. 6156–6166, 02 2019. [81] Kumar, Ranjeet; A. Kumar; and G. K. Singh. (2015) \"Electrocardiogram signal compression based on singular value decomposition (SVD) and adaptive scanning wavelet difference reduction (ASWDR) technique\" AEU-International Journal of Electronics and Communications 69.12: 1810-1822. [82] Feng, Jun; et al. (2018) \"A Secure Higher-Order Lanczos-Based Orthogonal Tensor SVD for Big Data Reduction.\" IEEE Transactions on Big Data. [83] Chen Y et al (2018) Application of singular value decomposition algorithm to dimension-reduced clustering analysis of daily load profiles. Automat Electr Power Syst 42(3):105–111 [84] Kang M, Kim JM (2013) Singular value decomposition-based feature extraction approaches for classifying faults of induction motors. Mech Syst Signal Process 41(1–2):348–356 [85] A. Winursito, R. Hidayat, A. Bejo, and M. N. Y. Utomo, “Feature Data Reduction of MFCC Using PCA and SVD in Speech Recognition System,” 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE), 2018, pp. 1-6, doi: 10.1109/ICSCEE.2018.8538414 [86] Higham, J.E., Brevis, W. and Keylock, C.J. (2016) A Rapid Non-Iterative Proper orthogonal Decomposition Based Outlier Detection and Correction for PIV Data. Measurement Science and Technology, 27, Ariticle ID: 125303. https://doi.org/10.1088/0957-0233/27/12/125303 [87] Guillemot, V., Beaton, D., Gloaguen, A., Löfstedt, T., Levine, B., Raymond, N. and Abdi, H. (2019) A Constrained Singular Value Decomposition Method That Integrates Sparsity and Orthogonality. PLoS ONE, 14, e0211463. https://doi.org/10.1371/journal.pone.0211463 [88] Husson, F., Josse, J., Narasimhan, B. and Robin, G. (2019) Imputation of Mixed Data With Multilevel Singular Value Decomposition. Journal of Computational and Graphical Statistics, 28, 552-566. https://doi.org/10.1080/10618600.2019.1585261. [89] Asiedu, L., Adebanji, A., Oduro, F. and Mettle, F. (2016) Statistical Assessment of PCA/SVD and FFT-PCA/SVD on Variable Facial Expressions. British Journal of Mathematics & Computer Science, 12, 1-23. https://doi.org/10.9734/BJMCS/2016/22141 [29] Ka [90] Cao, L.J. (2006) Singular Value Decomposition Applied to Digital Image Processing. Arizona State University Polytechnic Campus, Mesa, 1-15. [91] Santos, A.R., Santos, M.A., Baumbach, J., Mcculloch, J.A., Oliveira, G.C., Silva, A. and Azevedo, V. (2011) A Singular Value Decomposition Approach for Improved Taxonomic Classification of Biological Sequences. BMC Genomics, 12, Artticle No. S11. https://doi.org/10.1186/1471-2164-12-S4-S11 [92] Lassiter, A. (2013) Handwritten Digit Classification and Reconstruction of Marred Images Using Singular Value Decomposition. Virginia Tech. [93] Meng, C., Zeleznik, O.A., Thallinger, G.G., Kuster, B., Gholami, A.M. and Culhane, A.C. (2016) Dimension Reduction Techniques for the Integrative Analysis of Multi-Omics Data. Briefings in Bioinformatics, 17, 628-641. https://doi.org/10.1093/bib/bbv108 [94] Li, X., Ng, M.K., Ye, Y., Wang, E.K. and Xu, X. (2017) Block Linear Discriminant Analysis for Visual Tensor Objects with Frequency or Time Information. Journal of Visual Communication and Image Representation, 49, 38-46. https://doi.org/10.1016/j.jvcir.2017.08.004 [95] L. J. Cao, K. S. Chua, W. K. Chong, H. P. Lee, and Q. M. Gu, “A comparison of PCA, KPCA and ICA for dimensionality reduction in support vector machine,” Neurocomputing, vol. 55, no. 1, pp. 321–336, 2003, doi: https://doi.org/10.1016/S0925-2312(03)00433- 8. [96] Tharwat, A., Gaber, T., Ibrahim, A. and Hassanien, A.E. (2017) Linear Discriminant Analysis: A Detailed Tutorial. AI Communications, 30, 169-190. https://doi.org/10.3233/AIC-170729 [97] Fisher, E.M. (1936) Linear Discriminant Analysis. Statistics & Discrete Methods of Data Sciences, 392, 1-5. [98] C. O. S. Sorzano, J. Vargas, and A. P. Montano, “A survey of dimensionality reduction techniques,” pp. 1–35, 2014. [99] Park, C.H. and Park, H. (2008) A Comparison of Generalized Linear Discriminant Analysis Algorithms. Pattern Recognition, 41, 1083-1097. https://doi.org/10.1016/j.patcog.2007.07.022 [100] Yu, H. and Yang, J. (2001) A Direct LDA Algorithm for High-Dimensional Data—With Application to Face Recognition. Pattern Recognition, 34, 2067-2070. https://doi.org/10.1016/S0031-3203(00)00162-X [101] Zhang, T., Fang, B., Tang, Y. Y., Shang, Z. and Xu, B. (2010) Generalized Discriminant Analysis: A Matrix Exponential Approach. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 40, 186-197. https://doi.org/10.1109/TSMCB.2009.2024759 [102] Kedadouche, M., Liu, Z. and Thomas, M. (2018) Bearing Fault Feature Extraction Using Autoregressive Coefficients, Linear Discriminant Analysis and Support Vector Machine under Variable Operating Conditions. Applied Condition Monitoring, 9, 339-352. https://doi.org/10.1007/978-3-319-61927-9_32 [103] Xiong, H., Cheng, W., Hu, W., Bian, J. and Guo, Z. (2017) FWDA: A Fast Wishart Discriminant Analysis with its Application to Electronic Health Records Data Classification. 1-15. http://arxiv.org/abs/1704.07790 [104] Wu, L., Shen, C.H. and van den Hengel, A. (2017) Deep Linear Discriminant Analysis on Fisher Networks: A Hybrid Architecture for Person Re-Identification. Pattern Recognition, 65, 238-250. https://doi.org/10.1016/j.patcog.2016.12.022 [105] Krasoulis, A., Nazarpour, K. and Vijayakumar, S. (2017) Use of Regularized Discriminant Analysis Improves Myoelectric Hand Movement Classification. 2017 8th International IEEE/EMBS Conference on Neural Engineering (NER), Shanghai, 25-28 May 2017, 395-398. https://doi.org/10.1109/NER.2017.8008373 [106] Jusas, V. and Samuvel, S.G. (2019) Classification of Motor Imagery Using a Combination of User-Specific Band and Subject-Specific Band for Brain-Computer Interface. Applied Sciences (Switzerland), 9, 1-17. https://doi.org/10.3390/app9234990 [106] Wilson, S.R., Close, M.E. and Abraham, P. (2018) Applying Linear Discriminant Analysis to Predict Groundwater Redox Conditions Conducive to Denitrification. Journal of Hydrology, 556, 611-624. https://doi.org/10.1016/j.jhydrol.2017.11.045 [107] K. Torkkola. Linear discriminant analysis in document classification. In IEEE TextDM 2001, pages 800–806, 2001. [108] R. Haeb-Umbach and H. Ney. Linear discriminant analysis for improved large vocabulary continuous speech recognition. In IEEE International Conference on Acoustics, Speech, and Signal Processing, 1992, volume 1, pages 13–16, 1992. [109] Dimensionality Reduction Eisaku Maeda Tokyo Denki University, Tokyo, Japan (1) © Springer Nature Switzerland AG 2020 K. Ikeuchi, Computer Vision [110] R. Gutierrez-Osuna, “Dimensionality rereduction-lda.” Introduction to Pattern Recognition,Wright State University. [111] S. Raschka, “Linear discriminant analysis bit by bit.” Blog, August 2014 [112] Think India Journal Issn: 0971-1260 Vol-22-Issue-16-August-2019. A Comparative Study on Linear Dimensionality Reduction Techniques For Big Data Analytics Devinder Kaur, Oshin Bhardwaj, Surbhi Sharma, [113] C. Li, Y. Shao, W. Yin, and M. Liu, “Robust and sparse linear discriminant analysis via an alternating direction method of multipliers,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–12, 2019. [114] Y. Liu, R. Zhang, F. Nie, X. Li, and C. Ding, “Supervised dimensionality reduction methods via recursive regression,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–11, 2019. [115] S. Feng and H. Wang, “Comparison of PCA and LDA Dimensionality Reduction Algorithms based on Wine Dataset,” 2021 33rd Chinese Control and Decision Conference (CCDC), 2021, pp. 2791-2796, doi: 10.1109/CCDC52312.2021.9602325. [116] X. Liu, H. Xiong, and N. Shen, “A hybrid model of VSM and LDA for text clustering,” 2017 2nd IEEE International Conference on Computational Intelligence and Applications (ICCIA), 2017, pp. 230- 233, doi: 10.1109/CIAPP.2017.8167213. [117] S. Ji and J. Ye, “Generalized Linear Discriminant Analysis: A Unified Framework and Efficient Model Selection,” in IEEE Transactions on Neural Networks, vol. 19, no. 10, pp. 1768-1782, Oct. 2008, doi: 10.1109/TNN.2008.2002078. [118] R. Fisher, “The statistical utilization of multiple measurements,” Ann.Eugenics, vol. 8, no. 4, pp. 376– 386, Aug. 1938. [119] D. Zhang, Y. Zhao, M. Du, A new supervised dimensionality reduction algorithm using linear discriminant analysis and locality preserving projection, WSEAS TRANSACTIONS on INFORMATION SCIENCE and APPLICATIONS, E-ISSN (2013) 2224–3402. [120] Z. Liu, G. Liu, J. Pu, X. Wang, H. Wang, Orthogonal sparse linear discriminant analysis, International Journal of Systems Science (2018) 1–11. [121] M. Zhu, A. M. Martinez, Subclass discriminant analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (8) (2006) 1274–1286. [122] N. Gkalelis, V. Mezaris, I. Kompatsiaris, Mixture subclass discriminant analysis, IEEE Signal Processing Letters 18 (5) (2011) 319–322. [123] R. Ran, B. Fang, X. Wu, S. Zhang, A simple and effective generalization of exponential matrix discriminant analysis and its application to face recognition, IEICE TRANSACTIONS on Information and Systems 101 (1) (2018) 265–268. [124] Y. Xin, Q. Wu, Q. Zhao, Q. Wu, Semi-supervised regularized discriminant analysis for eeg-based bci system, in: International Conference on Intelligent Data Engineering and Automated Learning, Springer, 2017, pp. 516–523. [125] J. Ye, R. Janardan, Q. Li, Two-dimensional linear discriminant analysis, in: Advances in neural information processing systems, 2005, pp. 1569–1576. [126] C.-N. Li, Y.-H. Shao, W.-J. Chen, N.-Y. Deng, Generalized two-dimensional linear discriminant analysis with regularization, arXiv preprint arXiv:1801.07426. [127] G. Baudat, F. Anouar, Generalized discriminant analysis using a kernel approach, Neural computation 12 (10) (2000) 2385–2404 [128] Zizhu Fan; Yong Xu; Zhang, D. ,\"Local Linear Discriminant Analysis Framework Using Sample Neighbors\", IEEE Transactions on Neural Networks, , On page(s): 1119 - 1132 Volume: 22, July 2011 [129] Tang, Y. and Rose, R. (2008) A Study of Using Locality Preserving Projections for Feature Extraction in Speech Recognition. 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, 31 March-4 April 2008, 1569-1572. https://doi.org/10.1109/ICASSP.2008.4517923 [130] Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314 [131] Hyvärinen A, Oja E (2000) Independent component analysis: algorithms and applications. Neural Netw 13(4–5):411–430 [132] A. Hyvärinen, J. Karhunen, and E. Oja. (2001). Independent Component Analysis. New York: Wiley. [133] C. O. S. Sorzano, J. Vargas, and A. P. Montano, “A survey of dimensionality reduction techniques,” pp. 1–35, 2014. [134] A. Tharwat, “Independent component analysis: An introduction,” Appl. Comput. Informatics, vol. 17, no. 2, pp. 222–249, Jan. 2021, doi: 10.1016/j.aci.2018.08.006. [135] Raji Ramachandran, Gopika Ravichandran and Aswathi Raveendran, “Evaluation of Dimensionality Reduction Techniques for Big Data,” 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC), 23 April 2020 [136] Huang QH, Wang S, Liu Z (2007) Improved algorithm of image feature extraction based on independent component analysis. Opto-Electron Eng 1:121–125 [137] Yuen PC, Lai J-H (2002) Face representation using independent component analysis. Pattern Recogn 35(6):1247–1257 [138] Wang J, Chang C-I (2006) Independent component analysis-based dimensionality reduction with applications in hyperspectral image analysis. IEEE Trans Geosci Remote Sens 44(6):1586–1600. [139] Jing Wang and Chein-I Chang, “Independent component analysis-based dimensionality reduction with applications in hyperspectral image analysis,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 44, no. 6, pp. 1586-1600, June 2006, doi 10.1109/TGRS.2005.863297. [140] T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991. [141] Scholz, M., Gatzek, S., Sterling, A., Fiehn, O. and Selbig, J. (2004) Metabolite Fingerprinting: Detecting Biological Features by Independent Component Analysis. Bioinformatics, 20, 2447-2454. https://doi.org/10.1093/bioinformatics/bth270 [142] Pochet, N., De Smet, F., Suykens, J.A.K. and De Moor, B.L.R. (2004) Systematic Benchmarking of Microarray Data Classification: Assessing the Role of Non-Linearity and Dimensionality Reduction. Bioinformatics, 20, 3185-3195 [143] Hyvärinen, A. (1999) Sparse Code Shrinkage: Denoising by Nonlinear Maximum Likelihood Estimation. Neural Computation, 11, 1739-176 [144] Amari, S.-I., Cichocki, A. and Yang, H.H. (1996) A New Learning Algorithm for Blind Source Separation. In: Mozer, M., Ed., Advances in Neural Information Processing System, Morgan Kaufmann Publishers, Massachussets, 757-763. https://papers.nips.cc/paper/1115-a-new-learning-algorithm-for-blind-signal-separ ation.pdf [145] Hyvärinen, A. and Oja, E. (1997) A Fast Fixed-Point Algorithm for Independent Component Analysis. Neural Computation, 9, 1483-1492. https://doi.org/10.1162/neco.1997.9.7.1483

Copyright

Copyright © 2026 Daljeet Kaur Khanduja, Surjeet Kaur . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.